Apparatus and method for providing augmented reality service using sound

ABSTRACT

A method and an apparatus are provided for providing additional information service in a mobile communication terminal. A microphone receives a sound signal including an audible frequency band and an inaudible frequency band. Additional information related to the service is detected in the inaudible frequency band included in the sound signal. The detected additional information is extracted from the sound signal. Data for providing the service is acquired based on the extracted additional information. A service screen is displayed based on the acquired data.

PRIORITY

This application is a Continuation Application of U.S. Pat. No.8,749,584, issued on Jun. 10, 2014, which claims priority under 35U.S.C. §119(a) to a Patent Application filed in the Korean IntellectualProperty Office on Jul. 22, 2010 and assigned Serial No.10-2010-0070994, the entire disclosures of which are incorporated hereinby reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates generally to a mobile communicationterminal, and, more particularly, to an apparatus and method forproviding an augmented reality service using a sound in a mobilecommunication terminal.

2. Description of the Related Art

At a time of growing expansion of various services and additionalfunctions for mobile communication terminals, various service systemsproviding various types of information to users have been developed. Forexample, recently introduced systems can provide basic positioninformation of a mobile communication terminal when its user desires toknow his current position requests his position information, i.e.,information relating to the current position of the mobile communicationterminal, using the mobile communication terminal that he carries withhim.

One of the conventional techniques for providing such positioninformation in the augmented reality field may detect a position of auser with a Global Positioning System (GPS) and a gyroscope, and providea real image merged with its additional information to the user on asee-through Head-Mounted Display (HMD). This is the latest techniquethat can provide more useful information by mixing the real world withthe cyber world created by a computer, for the visual representation andinformation provision impossible in the real world. This technique iswidely known as an Augmented Reality (AR) technique, which can be used,for example, to provide a variety of information, including informationabout nearby attractions, to tourism travelers.

As described above, the conventional AR technique may provide additionalinformation associated with a real image in a current position. However,the additional information in the current position may not be providedwhen the real image cannot be acquired for some reason. Thus, if itwould be possible to provide an AR service, even in places where realimages cannot be acquired or their features cannot be detected, itsutilization may be enhanced.

SUMMARY OF THE INVENTION

An aspect of the present invention is to substantially solve at leastthe above problems and/or disadvantages and to provide at least theadvantages below. Accordingly, an aspect of the present invention is toprovide an apparatus and method for providing an Augmented Reality (AR)service even in a location in which a real image cannot be acquired.

Another aspect of the present invention is to provide an apparatus andmethod for providing an AR service using a sound instead of a realimage.

And yet another aspect of the present invention is to provide anapparatus and method for providing AR-based additional information.

According to one aspect of the present invention, a method is providedfor providing additional information service in a mobile communicationterminal. A microphone receives a sound signal including an audiblefrequency band and an inaudible frequency band. Additional informationrelated to the service is detected in the inaudible frequency bandincluded in the sound signal. The detected additional information isextracted from the sound signal. Data for providing the service isacquired based on the extracted additional information. A service screenis displayed based on the acquired data.

According to another aspect of the present invention, an apparatus isprovided for providing additional information service. The apparatusincludes a microphone for receiving a sound signal including an audiblefrequency band and an inaudible frequency band. The apparatus alsoincludes a controller for detecting the additional information relatedto the service in the inaudible frequency band included in the soundsignal, extracting the detected additional information from the soundsignal received through the microphone, acquiring data for providing theservice based on the extracted additional information, and configuring aservice screen based on the acquired data. The apparatus furtherincludes a display unit for displaying the service screen.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other aspects, features and advantages of the presentinvention will become more apparent from the following detaileddescription when taken in conjunction with the accompanying drawing inwhich:

FIG. 1 is a diagram illustrating a configuration of a system forproviding an AR service according to an embodiment of the presentinvention;

FIG. 2 is a block diagram of a mobile communication terminal forproviding an AR service according to an embodiment of the presentinvention;

FIG. 3 is a diagram showing an example of an AR service according to anembodiment of the present invention; and

FIG. 4 is a flowchart illustrating a method of providing an AR serviceaccording to an embodiment of the present invention.

DETAILED DESCRIPTION OF EMBODIMENTS OF THE PRESENT INVENTION

Embodiments of the present invention will be described herein belowreferring to the accompanying drawings. Like reference numbers andsymbols are used to refer to like elements through at the drawings. Inthe following description, well-known functions or constructions are notdescribed in detail since they would obscure the invention withunnecessary detail.

An apparatus and method for providing an Augmented Reality (AR) serviceusing sound are described herein. In order to accomplish this, thepresent invention includes receiving a sound signal if a function ofproviding the AR service in a mobile communication terminal is initiatedby a user, analyzing the received sound signal, determining whether thesound signal is carrying additional information, extracting theadditional information if the additional information is carried on thesound signal, acquiring data associated with the extracted additionalinformation, and providing the AR service using the acquired data.Accordingly, the present invention can not only provide various types ofadditional information in the current position but also the AR servicewithout the limitation of a location.

A system configuration for providing an AR service according to anembodiment of the present invention will now be described referring toFIG. 1.

Referring to FIG. 1, the system for providing an AR service may includea sound source device 110 for providing a sound signal, and a mobilecommunication terminal 100 for receiving the sound signal.

The sound source device 110 may insert additional information into asound signal according to a preset encoding scheme and transmit thesound signal. The sound source device 110 may broadcast through aspeaker 120 the sound signal on which the additional information iscarried. The sound source device 110 may be any device installed in apredetermined location, such as a subway station and a building. Thesound signal may be a signal in an audible frequency band that a usercan hear, and the additional information carried on the sound signal maybe a non-sound signal in an inaudible frequency band that the usercannot hear so that the additional information is not heard by a user asnoise. Accordingly, even if the additional information is carried on thesound signal, the user may not recognize any difference from a normalsound when the user hears a sound corresponding to the sound signal.

The mobile communication terminal 100 may start receiving the soundsignal through a microphone when an AR service function is activated bythe user. The mobile communication terminal 100 may determine whetheradditional information is carried on the sound signal, and if additionalinformation is carried on the sound signal, the mobile communicationterminal 100 may decode and extract the additional information carriedon the sound signal. After extracting the additional information bydecoding, the mobile communication terminal 100 may search for dataassociated with the additional information by comparing the additionalinformation with information stored in a its database. Thereafter, themobile communication terminal 100 may configure an AR service screenbased on the searched data and display the screen. The mobilecommunication terminal 100 may output voice through its speaker toprovide the AR service using the searched data.

Instead, the mobile communication terminal 100 may transmit theadditional information extracted from the sound signal to a server (notshown), receive AR data corresponding to the additional information fromthe server, and configure an AR service screen using the AR data.

The data associated with the additional information may includeinformation in a current position, such as associated images, a map, andlocal area information in the current position, and if the sound signalcorresponds to music, the data associated with the additionalinformation may include information associated with music. As describedabove, the additional information may be associated information in acurrent position where the sound signal is received, or additionalinformation associated with a sound resource. On the other hand, themobile communication terminal 100 may search for data associated withthe additional information in using a database of an external server toprovide information.

Operations and components of the mobile communication terminal 100 inthe above-described system will now be described in detail referring toFIG. 2. In FIG. 2, the mobile communication terminal 100 may include asound signal processor 200, a MICrophone (MIC) 210, a SPeaKer (SPK) 220,a controller 230, a display unit 240, and a memory 250. The mobilecommunication terminal 100 may further include a camera (not shown) forcapturing a real image to provide the AR service in which additionalinformation is added to the real image by using the additionalinformation carried on a sound signal, and a communication unit (notshown) for transmitting the additional information extracted from thesound signal to an external server and receiving AR data correspondingto the transmitted additional information from the external server.

The sound signal processor 200 may receive a sound signal through themicrophone 210, amplify the sound signal, and deliver the amplifiedsound signal to the controller 230. The microphone 210 may be activatedto receive a sound signal when the user initiates the AR servicefunction. The sound signal processor 200 may be connected to the speaker220 so that a sound signal for the AR service output from the controller230 is output to the outside.

The display unit 240 may generally display operation states and aplurality of pieces of information of the mobile communication terminal100 under control of the controller 230. When the user sets or activatesa necessary function, the display unit 240 may allow the user torecognize a set or activated state under control of the controller 230.The display unit 240 may display an AR service screen. As such, thedisplay unit 240 may display data associated with additional informationin a location at which the user is located, using graphic technology.For example, the user may view an image in a current position, whichdynamically varies according to additional information carried on areceived sound signal, together with data associated with the additionalinformation, and if the sound signal corresponds to music, the user mayview detailed information regarding the music.

The image in the current position may be an image included in theadditional information or a real image, which was captured by thecamera. If the image is the real image captured by the camera, themobile communication terminal 100 may determine the current position andcurrent bearing using a GPS and a gyroscope, and display informationregarding surroundings based on the additional information (e.g., menuinformation of a restaurant) carried on a sound signal together with thereal image.

The memory 250 may include Read Only Memory (ROM) and Random AccessMemory (RAM) for storing a plurality of programs and data. The memory250 may store data associated with additional information extracted froma sound signal. In other words, the memory 250 may include a databasefor storing data for providing the AR service.

The controller 230 may control the overall operation of the mobilecommunication terminal 100. According to an embodiment of the presentinvention, the controller 230 may determine whether additionalinformation carried on a sound signal is received, and search thedatabase for data corresponding to the additional information. Then, thecontroller 230 may configure an AR service screen based on the searcheddata, and output the configured AR service screen. Features may beinserted into the sound signal to make it possible to identify aposition of the additional information in the sound signal. The featuresmay be inserted in the form of specific codes or blanks. Accordingly,the controller 230 may detect the features inserted in the sound signaland extract the additional information between the detected features.

The additional information may have the following formats according tothe present invention. The additional information may be metadata orhave a link information format used to search the database for theposition of associated data. If the additional information is in ametadata format, the controller 230 may provide the AR service with onlythe metadata and output detailed information by searching the databasefor content associated with the metadata. As described above, themetadata may be used to provide various types of information withrespect to a received sound signal in connection to the database. Forexample, upon receipt of a sound signal on which various kinds ofinformation associated with music as additional information, such as agenre, lyrics, and sale information, are carried, the controller 230 mayobtain information useful to provide the AR service by analyzing thesound signal.

A process of providing the AR service will now be described referring toFIG. 3. The present invention may include extracting additionalinformation from a sound signal 300 on which the additional informationis carried, searching for data 320 associated with the extractedadditional information from a database 310, and providing the AR servicebased on the searched data.

An AR service screen made by the above-described process is provided toa user as shown in FIG. 3. Referring to FIG. 3, an AR service screen maybe provided on the display unit 240 when the mobile communicationterminal receives a subway announcement while it user is in a subwaytrain.

For example, when the user wants to receive the AR service in a currentposition while hearing the subway announcement, the user may activatethe AR service function of the mobile communication terminal 100. Inthat case, the mobile communication terminal 100 may start receiving thesubway announcement through the microphone 210. In accordance with anembodiment of the present invention, the subway announcementcorresponding to a sound signal may be an announcement on whichadditional information is carried. Thus, when the user receives suchsubway announcement on which additional information is carried, themobile communication terminal 100 may display various kinds ofinformation, such as exit information and surrounding restaurantinformation, together with an associated image in the current position.

Although in FIG. 3, the exit information is added to the associatedimage in the current position when the exit information is selected,restaurant location information may be added to the associated imagewhen the surrounding restaurant information is selected. That is,various changes may be made in an AR service screen. As described above,the content of a screen for the AR service may be provided in variousforms.

Although an AR service relating to public transportation, such as asubway train, is shown in FIG. 3, this is a mere example and the subjectmatter of the present invention are not limited thereto. For example,when a user desires to obtain further information regarding music heardwhile walking on the road, the mobile communication terminal 100 mayacquire additional information about the music, and display dataassociated with the music. Similarly, the mobile communication terminal100 may receive a sound source from a music vending machine and displayadditional information about the sound source to the user. Even thoughthe user does not acquire entire sound source associated with the fullmusic, the user may sufficiently obtain desired additional informationwith only a part of the sound source. Additionally, according to anembodiment of the present invention, the mobile communication terminal100 may provide detailed information about a product to the user bymerely receiving an announcement in a specific location, e.g., a store.As described above, additional information may be carried on varioussound signals depending on selections made by service providers.

FIG. 4 is a flowchart illustrating a method of providing an AR serviceusing a sound signal in the mobile communication terminal 100, accordingto an embodiment of the present invention. Although a method ofextracting additional information by searching for features will beillustrated in the following description, the method of extractingadditional information according to an embodiment of the presentinvention is not limited thereto.

Referring to FIG. 4, upon starting the AR service function, the mobilecommunication terminal 100 receives a sound signal through microphone210 in step 400. In step 410, the mobile communication terminal 100determines whether the sound signal is carrying additional information.If the sound signal is carrying additional information, the mobilecommunication terminal 100 detects features for extracting theadditional information from the sound signal by decoding the soundsignal according to a preset decoding scheme, in step 420. The detectedfeatures may be used to identify a boundary of the sound signal on whichthe additional information is carried. Accordingly, in step 430, themobile communication terminal 100 extracts the additional informationfrom the sound signal based on the detected features. In step 440, themobile communication terminal 100 searches the database for datacorresponding to the extracted additional information and acquires thesearched data. In step 450, the mobile communication terminal 100provides the AR service based on the acquired data.

As described above, the additional information carried on a sound signalenables the content associated with the sound signal to be output.Accordingly, a user may receive various kinds of information associatedwith a sound heard in a current position in various forms, such asimages, graphics, and voice, in real-time.

As apparent from the foregoing description, the present invention mayprovide AR-based additional information using a sound instead of a realimage, making it possible to provide various kinds of additionalinformation even in a location, such as a subway train and a building,where no specific features may be detected, or real images may not beobtained because of the difficulty in taking a photograph. Additionally,the present invention may extract additional information from a soundreceived in a current position, thereby providing a map and touristinformation in the current position and contributing to an expectedincrease in use of various additional services.

While the invention has been shown and described with reference tocertain embodiments thereof, it will be understood by those skilled inthe art that various changes in form and details may be made thereinwithout departing from the spirit and scope of the invention as definedby the appended claims and their equivalents.

What is claimed is:
 1. A method for providing additional informationservice in a mobile communication terminal, the method comprising:receiving, through a microphone, a sound signal including an audiblefrequency band and an inaudible frequency band; detecting additionalinformation related to the service in the inaudible frequency bandincluded in the sound signal; extracting the detected additionalinformation from the sound signal; acquiring data for providing theservice based on the extracted additional information; and displaying aservice screen based on the acquired data.
 2. The method of claim 1,wherein the extracting of the detected additional information from thesound signal comprises: decoding the sound signal according to a presetdecoding scheme; and extracting the detected additional information fromthe decoded sound signal.
 3. The method of claim 1, wherein the acquireddata includes at least one of an associated image, a map, andsurrounding information in a position where the sound signal isreceived.
 4. The method of claim 1, wherein the acquired data includesmusic-associated information.
 5. The method of claim 1, whereinacquiring the data for providing the service comprises searching adatabase for data associated with the extracted additional information.6. The method of claim 1, wherein the acquiring of the data forproviding the service comprises: transmitting the extracted additionalinformation to a server; and receiving, from the server, data forconfiguring the service screen corresponding to the transmittedadditional information.
 7. The method of claim 1, wherein the additionalinformation associated with the service includes a non-sound signal. 8.The method of claim 1, wherein the receiving of the sound signalcomprises activating the microphone when additional information servicefunction is activated by a user.
 9. An apparatus for providingadditional information service, the apparatus comprising: a microphonefor receiving a sound signal including an audible frequency band and aninaudible frequency band; a controller for detecting the additionalinformation related to the service in the inaudible frequency bandincluded in the sound signal, extracting the detected additionalinformation from the sound signal received through the microphone,acquiring data for providing the service based on the extractedadditional information, and configuring an service screen based on theacquired data; and a display unit for displaying the service screen. 10.The apparatus of claim 9, wherein the controller decodes the soundsignal carrying the additional information, according to a presetdecoding scheme and extracts the detected additional information fromthe decoded sound signal.
 11. The apparatus of claim 9, wherein theacquired data includes at least one of an associated image, a map, andsurrounding information in a position where the sound signal isreceived.
 12. The apparatus of claim 9, wherein the acquired dataincludes music-associated information.
 13. The apparatus of claim 9,further comprising: a database for storing the data for providing theservice.
 14. The apparatus of claim 9, wherein the controller transmitsthe extracted additional information to a server, receives datacorresponding to the transmitted additional information from the server,and configures the service screen based on the received data.
 15. Theapparatus of claim 9, wherein the additional information associated withthe service includes a non-sound signal.
 16. The apparatus of claim 9,wherein the controller activates the microphone when a user activatesadditional information service function and receives the sound signalthrough the microphone.